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Multiple Response System (MRS) Evaluation Report to 
the North Carolina Division of Social Services (NCDSS) 

Executive Summary 



I n response to a request from the North 
Carolina Division of Social Services 
(DSS),Center for Child and Family Policy at 
The Terry Sanford Institute at Duke University 
evaluated the Multiple Response System reform 
for families reported for child maltreatment. The 
North Carolina State Legislature has mandated 
the evaluation of five key aspects of the Multiple 
Response System (MRS) as field-tested by 10 
County Departments of Social Services (DSS) 
in North Carolina, specifically: Child Safety; 
Timeliness of Response; Timeliness of 
Service; Coordination of Local Human 
Services; and Cost-Effectiveness. 

Data for this evaluation were assembled from 
State data systems and through original data 
collection that included Child Protective Services 
(CPS) Reports, Services Information System (SIS) 
Data, surveys of family members, and focus 
groups. 

The methods used to complete the evaluation 
included tracking of changes in the 10 pilot MRS 
counties over time and contrasting the activities 
and outcomes in the 10 MRS counties with 
matched counties that had not yet implemented 
MRS. Each MRS county was matched to a control 
county based on similarities in the following 
criteria: total population; child population; 
reported rates of child maltreatment - both 
investigated and substantiated; rates of 
children in DSS custody; and rates of children 
in foster care for the first time. Because 



Mecklenburg County does not have an 
appropriate comparison county due to 
substantial differences from other counties in 
population size, investigation rates, and 
substantiation rates, Mecklenburg was evaluated 
only through comparisons to itself over time and 
was not included in analyses that involve control 
counties. The following conclusions were drawn 
from the evaluation: 

Safety 

MRS has not adversely affected the level of 
children’s safety in the brief period of its 
implementation, according to official records of 
investigations and substantiations of child 
maltreatment. 

Timeliness of Response 

MRS has not significantly altered the likelihood 
that families will receive an initial response from 
DSS within 72 hours or the likelihood that they 
will receive an official case decision within 30 
days. There was, however, a trend toward 
slightly increased initial response time in MRS 
counties. Response times in 2004 should be 
closely monitored to determine whether or not 
this trend continues. 

Timeliness of Service 

MRS has not altered the median length of time 
from report to the initiation of services or the 
proportion of families that receive initial 
services within 30 days of a report. 
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Coordination of Local Human Services 

MRS allows for better coordination and 
communication across agencies. The majority of 
families in the family assessment track of MRS 
counties were very satisfied with the way that their 
family was treated by the CPS worker and with 
the help that they received. In response to questions 
about their experience with staff of County DSS, 
families’ comments included: 

-- “If child services had not been involved I don’t 
know what I would have done. They bent over 
backwards to help mj family. ” 

- “CPS has helped my family to the fullest. They 

helped me build up my confidence in all areas. ” 

- “My social worker not only did her job well, but 

also cared about us and our problems. She did 
everything possible to make sure we could pull 
through our hard time. ” 

- ‘I just wanted to say that thy gave me a chance 

and now I made it finally. ” 

- ‘I think these people are here to help children and 

to help parents realise kids come first. ” 

Social workers and supervisors unanimously 
agreed that the Multiple Response System is a good 
way to serve families that allegedly maltreat their 
children. Workers appreciated the fact that MRS 
allows them to be more respectful of families and 
to take into consideration the whole picture 
instead of concentrating only on the specific 
alleged incident. 

Cost-Effectiveness 

A comprehensive cost-effectiveness analysis was 
not feasible. In the short run, aU MRS counties 
were able to re-allocate staff members and 
resources to accommodate the needs of the 
Multiple Response System without additional 
funds or a change in turnover rates. However, 
staff members experienced increased challenges 
in managing cases while working with new reports, 
leading to increased stress levels. It is possible that 
increased stress will lead to increased staff 
turnover or declining quality of service over time; 
however, better training and staff support might 
mitigate this stress. 



Recommendations 

The following are recommendations based upon 
the findings of this report and feedback received 
through the focus groups in the nine counties 
visited: 

- Limit caseload size to six to eight families per 

worker or implement a “team model” with 
teams consisting of investigators/ assessors, 
case managers, and foster care workers that 
act as a cohesive unit to serve a family. 

- Implement additional and consistent training 

for line staff. 

- Implement supervisory and management 

training immediately. 

- Appropriate state dollars to conduct cross- 

agency family-centered training for all child- 
serving agencies. 

- Create a new decision category to address those 

instances when families receive services 
during Assessment period and do not need 
additional supervision beyond this time, 
called “services received, no further services 
recommended.” 

- Conduct a more systematic and detailed 

evaluation when MRS has been in practice 
for a longer period of time. 

- Evaluate the next wave of implementation 

immediately. 
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Multiple Response System (MRS) Evaluation Report to 
the North Carolina Division of Social Services (NCDSS) 

Introduction 



Purpose 

The purpose of this report is to evaluate five key 
aspects of the Multiple Response System (MRS) 
as field-tested by ten County Departments of 
Social Services (DSS) in North Carolina. In 
agreement with the North Carolina Division of 
Social Services (NCDSS) and mandated by 
Senate Bill 1005 Section 21.46 (b), this 
evaluation includes data collection and analyses 
to address the following aspects of MRS 
implementation: 

(1) Child Safety 

(2) Timeliness of Response 

(3) Timeliness of Service 

(4) Coordination of Local Human 
Services 

(5) Cost Effectiveness 

Evaluator 

In response to a request from the NC Division 
of Social Services (DSS), the Center for Child 
and Family Policy at The Terry Sanford Institute 
at Duke University has engaged in the evaluation 
of the Multiple Response System to families 
reported for child maltreatment. 

The Center for Child and Family Policy brings 
together scholars, policymakers, and practitioners 
to solve problems facing children in contemporary 
society by undertaking rigorous social science 
research and then translating important findings 
into policy and practice. The Center is currently 



addressing issues of early childhood adversity, 
education policy reform, and youth violence and 
problem behaviors. Researchers at the Center 
design interventions for youth, implement them 
in school and community settings, and evaluate 
them through rigorous designs and sophisticated 
analytic methods. Center researchers and staff 
also work closely with families of at-risk children 
to implement and evaluate programs designed to 
foster healthy family dynamics. 

Kenneth A. Dodge, Ph.D, who has served as the 
Principal Investigator for this evaluation, is the 
William McDougall Professor of Public Policy 
and Professor of Psychology and the Director of 
the Center of Child and Family Policy at Duke. 
For the past 25 years. Dr. Dodge has published 
over 200 scientific articles and has been the PI 
on research grants totaling over $30 million, 
several involving multi-site collaborations. He is 
the recipient of a Senior Scientist Award from 
the National Institute on Drug Abuse to study 
the development and prevention of drug use in 
youth. Most recently, he has been concerned with 
translating knowledge from prevention science 
into effective public policies for children, youth 
and their families. 

The evaluation team included staff members of 
the Center for Child and Family Policy with 
expertise in the areas of data management, 
statistics, project coordination, and program 
evaluation. Adele Spitz -Roth served as the project 
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coordinator for this evaluation. Spitz Roth has 
over 20 years of experience in program 
development and hands-on organizational, 
systems and project management in health and 
human services delivery systems. Christina 
Christopoulos, Ph.D., served as the Research 
Coordinator for this evaluation. For the past 1 3 
years. Dr. Christopoulos has coordinated the 
research component of the Fast Track multi-site 
conduct disorder prevention/intervention 
project. Katherine Rosanbalm, Ph.D., served as 
the statistician for this evaluation. Dr. Rosanbalm 
has worked as a program evaluator and statistician 
for numerous state and federally funded 
initiatives and research studies, including 
statewide pilot implementation of previous 
DHHS programs in North Carolina. Claire 
Osgood, assisted by Matt Edwards, was 
responsible for the data management and 
programming needs for this evaluation. Together, 
they have over 20 years of experience in data 
management, programming, and technical report 
writing. 

Background 

North Carolina’s Multiple Response System 
(MRS) began with a mandate by the North 
Carolina General Assembly (Session Law 2001- 
424, Senate Bill 1005, “Appropriations Act of 
the General Assembly”). This mandate required 
that the North Carolina Division of Social 
Services pilot an alternative response system for 
child protection with selected reports of 
suspected child neglect. Ten pilot counties were 
chosen to implement MRS, including: Alamance, 
Bladen, Buncombe, Caldwell, Craven, Franklin, 
Guilford, Mecklenburg, Nash, and Transylvania. 
These counties began preliminary 
implementation of MRS in 2002, and countywide 
implementation in January 2003. 

North Carolina’s Multiple Response System 
utilizes seven key strategies, including: 



/. M strengths-hased, structured intake process. 
Emphasis is placed on family strengths 
along with needs. Includes structured 
intake tools with consistent screening 
criteria for identifying child abuse, 
neglect, and dependency reports. 

2. A choice of two approaches to reports of child 
abuse, neglect, or dependencj. Allows a 
differential response to child neglect and 
dependency reports that provides a more 
tailored approach for each family, 
facilitating a partnership among local 
agencies and communities to address all 
needs of the child and family. An 
investigative assessment track is 
followed for cases requiring a forensic 
response, and a family assessment track 
is followed for dependency cases and 
selected cases of suspected neglect that 
might be better served by service 
delivery than by a forensic response. 

3. Coordination between law enforcement agencies 
and child protective services for the investigative 
assessment approach. County Departments 
of Social Services continue to work 
closely with law enforcement agencies, 
particularly in investigating and, when 
appropriate, prosecuting cases on the 
investigative track. The development of 
formal Memoranda of Agreement 
facilitates this process. 

4. A re-design of in-home services. Allows for 
a continuum of services of varying 
intensity depending on the needs of the 
family and the concerns for safety of the 
children. This continuum addresses the 
three core child outcomes of safety, 
permanence, and well-being. 

5. Implementation of Child and Family Team 
meetings during the provision of in-home 
services. Child and Family Team meetings 
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are used as a part of in-home services to 
bring all involved agencies and 
community/ family resources and 
supports to the table. A Child and Family 
Team (CFT) is a group of people that 
the parent and social worker together 
decide to invite to a meeting. The 
common threads of this group are that 
everyone knows the family (possibly in 
different contexts) and can honestly 
discuss the situation, identify needs, 
problem-solve, and reach consensus on 
a service plan. A Child and Family Team 
meeting is “with,” not “about,” the 
family. 

6. Implementation of shared-parenting meetings 
in child placement cases. When a child is 
placed in foster care, shared-parenting 
meetings are held within the first seven 
days for the social worker, birth parents, 
and foster parents to discuss the care of 
the child. 

7. Collaboration between the Work First Family 
Assistance and child welfare programs. CWS/ 
CPS work closely with Work First Family 
Assistance programs to provide families 
with financial, employment, and 
community services to help them become 
self-sufficient. 

The second strategy of the Multiple Response 
System allows counties to respond to reports of 
child maltreatment in one of two ways: an 
investigative assessment (traditional child 
protective services response) or a family 
assessment (alternative family-centered 
response). The investigative assessment, or 
forensic track, is applied to all reports that meet 
the definition of abuse or abandonment and 
special types of neglect reports. The family 
assessment track is applied to most reports of 
neglect or dependency, though social workers and 
supervisors may always choose to place a neglect 



or dependency case into the forensic track if they 
feel that the investigative assessment approach 
is needed to ensure the safety of the children. 
Both assessment approaches are family-centered 
and meet the needs of safety issues for children. 

The investigative assessment track, or forensic 
track, follows the traditional child protective 
services approach to child maltreatment reports. 
It typically involves an unannounced home visit, 
separate child interviews that may occur without 
the parents’ knowledge or consent, and 
interviews of collateral information sources 
without the parents’ knowledge. This track 
focuses on determining whether or not a specific 
reported case of child maltreatment has occurred. 
Following the investigative assessment, there are 
two possible findings: (1) substantiated, 
indicating that the reported incident occurred and 
child protective services are required, or (2) 
unsubstantiated, indicating that the reported 
incident cannot be proven, though services may 
be recommended if a need is identified. Again, 
this track continues to be utilized with reports 
of child abuse as well as the following special 
types of reports: 

- Abandonment 

- A child fatality when there are 
surviving children in the family 

- A child in custody of local DSS, family 
foster homes, residential facilities, child 
care situations, and reciprocal 
investigations 

- A child taken into protective custody 
by physician or law enforcement, 
pursuant to N.C. General Statue 7B-308 
& 500 

- The medical neglect of disabled infants 
with life threatening condition, pursuant to 
Public Law 98-457 (Baby Doe) 

-A child hospitalized (admitted to hospital) 
due to suspected abuse/neglect. 
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Additionally, any cases of child neglect may be 
assigned as an investigative assessment if the 
agency determines it is necessary to ensure child 
safety. 

The family assessment track provides an 
alternative approach to selected child 
maltreatment reports, including most neglect and 
all dependency cases. In specific cases, neglect 
allegations might follow the investigative 
assessment track. The family assessment track 
follows a strengths-based approach that attempts 
to engage the family in determining needs and 
finding solutions. By accessing extended family 
and community resources and facilitating a team 
approach to address identified needs, the family 
assessment track aims to stabilize the family and 
enable the parents to care for their children better. 
Initial interviews of parents and children are 
scheduled with the parents, parents are informed 
about collateral interviews, and no perpetrator 
is identified. This track focuses on total child 
well-being, assessing all of the family’s needs, 
rather than solely investigating a specific reported 
instance of neglect. Following a family 
assessment, there are three possible findings: (1) 
services needed, indicating that child protective 
services are required, (2) services 
recommended, indicating that services are 
voluntary but recommended, and (3) services 
not recommended, indicating that no service 
need has been identified. 

Method 

Selection of Comparison Counties 

Changes in the 10 pilot counties were tracked 
over time since 1996. These pilot counties were 
also contrasted with matched counties that have 
not yet implemented MRS.' Each pilot county 
was matched to a control county based on 
similarities in the following criteria: 



“ Total population 
“ Child population 

“ Reported rates of child maltreatment 
both investigated and substantiated 
“ Rates of children in DSS custody 
— Rates of children in foster care for the 
first time. 

Selection of the comparison counties also took 
into account geographic similarities and an 
understanding of the policies and practices of 
each county DSS. Selection decisions were 
reviewed and approved by the Division of Social 
Services. The pilot and control counties are listed 
as pairs in Table 1. Note that Mecklenburg 
County does not have a matched comparison 
county. Due to its size and very low rates of 
investigations and substantiations, there is no 
county in North Carolina than can be 
appropriately matched with Mecklenburg. 
Therefore, Mecklenburg is evaluated only 
through comparisons to itself over time, but is 
not included in analyses that involve control 
counties. 

Note that these two methods of evaluation 
(examination of counties’ trends over time and 
contrasts between pilot and matched-control 
counties) cannot provide a highly rigorous 
analysis of the effects of MRS because alternate 
interpretations of findings will always be 
plausible. It will always remain plausible that 
changes across time are due to some other 
important event (such as a change in the 
economy) rather than the introduction of a new 
system. Also, it will always remain plausible that 
the differences between the MRS counties and 
matched control counties are due to some factor 
that led the MRS counties to be selected in the 



' Some of the control counties will be implementing MRS in 
2004. However, none of them had started MRS as of December 
2003, and therefore MRS implementation does not overlap with 
the timeframe of this evaluation. 
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first place (such as their readiness for reform) 
rather than the MRS system. A true experiment 
with random assignment of counties would be 
needed to provide a more rigorous test of the 
effects of MRS. 



Table 1 — Matched Pilot and Control Counties 



Pilot 


Control 


Alamance 


Rowan 


Bladen 


Warren 


Buncombe 


Catawba 


Caldwell 


Surry 


Craven 


Cleveland 


Franklin 


Pender 


Guilford 


Durham 


Mecklenburg 

Nash 


Wayne 


Transylvania 


Jackson 



Data Sources 

Data for this evaluation were assembled from 
state data systems and through original data 
collection as described below. 

Child Protective Services (CPS) Reports 

The North Carolina Department of Health and 
Human Services (DHHS) collects all CPS reports 
of child abuse and neglect from each county. The 
data from these reports are stored in the Client 
Services Data Warehouse in the Central Registry. 
Data for 10 pilot counties and nine comparison 
counties were extracted from the Data Warehouse, 
providing information on individual children that 
included report and investigation dates, the type 
of maltreatment reported, whether the report was 
substantiated, and the type of maltreatment 
substantiated. See the Appendix for a detailed 
description of the CPS report data used in this 
evaluation. 



Services Information System (SIS) Data 

Like the CPS reports, DHHS provides SIS data 
via the Client Services Data Warehouse. These 
data include information on all social services 
provided to any person. Data for 10 pilot 
counties and nine comparison counties were 
extracted from the Data Warehouse, providing 
information on individual children that included 
service start and end dates as well as service 
codes and descriptions. See the Appendix for a 
detailed description of the SIS data used in this 
evaluation. 

Surveys of Family Members 

A core component of MRS is the family-centered 
approach. Therefore, feedback from families is 
crucial. To evaluate family satisfaction, original 
data were collected using an anonymous “sealed 
envelope approach”. This process of getting 
feedback and information from families and 
residents has worked in other settings and has 
successfully maintained the confidentiality of the 
responses.^ 

As a part of the sealed envelope approach, the 
social worker working with a family in the Family 
Assessment track interviewed a family member 
using the North Carolina Multiple Response 
System Family Involvement Survey (developed 
in partnership between the initial MRS counties 
and NCDSS). To ensure that literacy did not 
influence response, the social workers read the 
questions to the family member and the family 
member subsequently answered each question 
privately. Upon completion of the survey, the 
family member placed the survey in an envelope 
and sealed it and then placed the envelope in a 
larger envelope containing other unidentified 
sealed envelopes, thereby maintaining the 



^ Coulton, C.J., Korbin, J.E., & Su, M. (1999). Child Abuse and 
Neglect, 23, 1019-1040. 
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confidentiality of his/her responses. AU the sealed 
envelopes were subsequently sent to the Center 
for Child and Family PoUcy at Duke University 
for processing. Each county was asked to conduct 
25 sealed-envelope surveys with families who 
were involved in Child Protective Services 
through the Family Assessment/Case 
Management track between December 1 and 
December 31, 2003. Each family was given a $10 
Wal-Mart gift card for their participation. 

Focus Groups 

Focus groups with front-line workers and 
community partners were held to address 
questions of worker satisfaction and interagency 
coordination of services. Each county was visited 
by two evaluators who led the focus groups and 
recorded workers’ and community partners’ 
responses to a set of prescribed questions.^ 
Depending on department size, some counties had 
all their staff members participating and others 
had representatives participate. All the counties 
had a very high representation of staff members. 
Most counties had at least three community 
partners participate. 

In addition, focus groups were held with 
supervisors to address the utilization and re- 
allocation of resources. A prescribed set of 
questions was used to examine how resources were 
shifted within the agencies in order to implement 
MRS. Sample questions for both focus groups are 
provided in the Appendix. 

Findings 

The year 2003 brought major changes for the 10 
MRS counties in: (1) the DSS response 
alternatives about how to investigate cases (called 
“track”); and (2) the DSS decision alternatives 
(called “decisions”). Two response tracks are 
possible in MRS counties (called forensic track 
and family assessment track), whereas other 
counties have only one response track. Within 
the forensic track in MRS counties, one of five 



decisions must be reached: substantiated abuse, 
substantiated neglect, substantiated abuse and 
neglect, substantiated dependency, or 
unsubstantiated. In other counties, these five 
possible decisions apply to all cases. The MRS 
counties also have three decisions that are possible 
for cases in the family assessment track: services 
needed, services recommended, and services not 
recommended. If during the family assessment 
it is suspected that abuse or special categories of 
neglect that necessitate forensic investigation 
may have occurred, the case is then transferred 
to the forsenic track. 

Figure 1 depicts a flow chart of the possible 
response alternatives and decisions for MRS 
counties, along with the numbers and rates for 
each possible category, summed across the 10 
MRS counties. This figure shows that in MRS 
counties in 2003, 48.5% of the total of 23,016 
investigations were placed into the forensic track 
and 51.5% were placed into the family assessment 
track. Of the cases placed in the forensic track, 
4.6% were substantiated for abuse or abuse and 
neglect, 26.8% were substantiated for neglect, 
0.6% were substantiated for dependency, and 
68.0% were not substantiated. Of the cases 
placed in the family assessment track, 16.2% 
received a decision of services needed, 34.1% 
received a decision of services recommended, 
and 49.7% received a decision of services not 
recommended. The remainder of the findings 
addressed the five questions asked by the 
Legislature. 



^ The evaluators visited nine of the pilot counties between 
December 11, 2003 and January 14, 2004. 
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Figure 1: Distribution of Investigations and Case Decisions 
2003 Pilot MRS counties 




This chart includes all investigations, which may include multiple investigations for a single child. 

lA % = percentage of Investigative Assessments that fall into each decision category 

FA % = percentage of Family Assessments that fall into each decision category 

Total % = percentage of total investigations that fall within each track and decision category 

* This category includes both substantiated abuse and substantiated abuse and neglect 

Source: FindingFigI 



Child Safety 

The most critical issue in the evaluation of MRS 
implementation is the safety of children. It is 
important to evaluate whether the focus in MRS on 
a family-centered approach alters the likelihood that 
children are safe. Over several years, safety can be 
measured by examining trends in rates of child abuse 
and neglect, but, given that MRS was fully 
implemented for only one year through December 
31, 2003, the period of follow-up to examine child 
safety is short. As a consequence, any conclusions 
about child safety should be regarded with caution 
and should be considered as tentative. Because the 
MRS reform no longer determines whether the 
alleged incident of neglect occurred but rather 
determines whether the families are in need of 
services, the question of child safety can be 
addressed best by tracking investigation rates for 
all children and decisions about child abuse (but 
not neglect). 



Several strategies were followed in evaluating 
child safety using data from Child Protective 
Services reports stored in the Client Services Data 
Warehouse in the Central Registry. First, trends 
across years were examined within each of the 
pilot counties to determine whether rates of 
investigations and rates of substantiated child 
abuse changed significandy during 2003 from 
previous years. Figures 2, 3, and 4 provide graphs 
of the rates of children with investigations (per 
1,000 child population), children with 
substantiations of abuse (per 1,000 child 
population), and the percentages of children with 
investigations that resulted in substantiation of 
abuse, respectively, for the mean of the nine MRS 
counties, the mean of the nine matched-control 
counties, and Mecklenburg County, for each year 
from 1996 through 2003. 
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Figure 2: Rates of Investigations per 1,000 Children 

Across Time, by Type of County 




-♦—Control ■ Pilot a Mecklenburg 



Numbers based on children with investigations. source: saMyFig234.sas 



Figure 3: Rates of Substantiated Abuse per 1,000 Children 
Across Time, by Type of County 




Numbers based on children with abuse substantiations. 

Source: SafetyFig234.sas 
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Figure 4: Percent of Investigations that are Substantiations of Abuse 
Across Time, by Type of County 




-♦—Control ■ Pilot Mecklenburg 



Numbers based on children with investigations. source: safetyF/g234.sas 



These three figures show that rates of 
substantiated abuse did not increase or decrease 
in MRS counties with the initiation of MRS. For 
investigations in 2003, control and MRS counties 
substantiated abuse at the rates of 2.76% and 
2.61%, respectively. These rates are slightly 
lower than rates in the previous five years and 
do not significantly differ from one another, 
according to standard statistical tests (matched- 
pair t-test). According to official rates of 
investigations and substantiations of abuse, 
child safety was not altered due to the 
introduction of MRS. 

Another way to look at safety in the short term is 
to examine rates of re-investigations of children 
who had previously been investigated by DSS; in 
other words, the proportions of cases for which 
children returned to the system within six months. 
To investigate safety in this manner, the following 
annual rates of re-investigation for the MRS 
counties and the control counties for both 2003 
and the pre-MRS year of 2001 were examined: 



(1) the proportions of investigated children 
(including both forensic investigations and family 
assessments) who had a second investigation 
within six months after the first investigation; 
and (2) the proportions of children who had been 
substantiated as abused who had a second 
investigation within six months after the first 
investigation. These analyses were restricted to 
cases reported in the first six months of 2003, 
so that a six-month follow-up was possible by 
December 31, 2003. For comparison purposes, 
all cases investigated in the first six months of 
each preceding year were also followed up for 
six months. Figures 5 and 6 depict these two 
rates, respectively, for nine MRS counties, nine 
control counties, and Mecklenburg County. 
Because of the low numbers of children with 
substantiated abuse, particularly in smaller 
counties, proportions of children with abuse 
substantiations that are re-investigated are quite 
sensitive to small changes in the data. This 
unreliability results in the variability across years 
that is evident in Figure 6. 

WW 
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Figure 5: Percent of Children with an Investigation in the First Half of 
the Year that have a Subsequent Investigation within 6 Months 
Across Time, by Type of County 




Control ■ Pilot —a— Mecklenburg 



Numbers based on children with investigations. 



Figure 6: Percent of Children with a Substantiated Abuse in the First 
Half of the Year that have a Subsequent Investigation within 6 Months 
Across Time, by Type of County 




Numbers based on children with abuse substantiations. 
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When the 10 MRS counties (Mecklenburg 
included) were examined over time, no significant 
changes were seen in rates of re-investigation 
between 2001 and 2003. In 2003, of those 
children who had been investigated during the first 
half of the year, 13.4% were re-investigated within 
six months, as compared to 14.7% in 200R. This 
change is not statistically significant. Of those 
children with a substantiation of abuse in the first 
half of 2003, 7.4% were re-investigated within 
six months, as compared to 14.6% in 2001. This 
difference is not statistically significant. Again, the 
low numbers of children with substantiated abuse 
make these re-investigation rates quite variable, 
thus even this seemingly large difference over time 
may be due to chance. It is, however, promising 
that the re -investigation rate for children with 
substantiated abuse has declined over the past two 
years in MRS counties; this rate should be 
monitored for future trends. For more information 
on the analytic methodology and statistical 
findings, refer to the Appendix. 

Next, the rates of re-investigation across time in 
the nine MRS counties and their matched control 
counties were compared. For each county, the 
change in rate of re-investigation between 2001 
and 2003 was computed, and the average change 
for MRS counties was contrasted with the average 
change for control counties. The mean change in 
the rate of re-investigations (children with an 
investigation in the first half of the year who had 
another investigation within six months) was - 
1.8% in the MRS counties, and —2.5% in the 
control counties. For children with abuse 
substantiations in the first half of the year who 
had another investigation within six months, the 
MRS change rate from 2001 to 2003 was -8.5%, 
whereas the control county change rate was — 
3.2%. These differences were not statistically 
significant. These findings indicate that the 
introduction of a two-track system using MRS 
has not altered the level of safety for children 
who have been investigated or substantiated as 
abused, over the following six-month period. 



Within the 10 MRS counties, re-investigation 
rates were examined for family assessment and 
forensic tracks in 2003. These rates were not 
expected to be comparable, thus no statistical 
analysis was completed. Of children who had an 
assessment in the family assessment track during 
the first half of the year, 13% were re-investigated 
within six months. Of children who were 
investigated in the forensic track during the first 
half of the year\ 13.4% were re-investigated 
within six months. Again, 7.4% of children with 
an abuse substantiation (by definition, all were in 
the forensic track) in the first half of 2003 were 
re-investigated within six months. 

Timeliness of Response 

Timeliness of response was operationalized as 
both the amount of time taken to initiate an 
investigation following a report of maltreatment 
and the amount of time taken to reach a case 
decision. Data from Child Protective Services 
reports stored in the Client Services Data 
Warehouse in the Central Registry were used for 
all analyses pertaining to timeliness of response. 

Initial Response 

County Departments of Social Services are 
required to initiate a response within a maximum 
of 72 hours of receipt of an accepted report 
(dependent on the type of allegation). When a 
report is accepted for investigation, it is called a 
“case.” The proportions of all cases for which 
the county DSS did initiate a response within the 
required 72-hour period are depicted in Figure 7, 
for each year from 1996 through 2003 for the mean 
of the nine MRS counties, the mean of the nine 
control counties, and Mecklenburg County. 



These percentages include data from all 10 MRS counties in- 
eluding Mecklenburg, thus may differ from the percentages in 
Figures 5 and 6. 

^ Children who had both a forsenic investigation and a family 
assessment during the first half of the year are included in both 
numbers. 



Center for Child and Family Policy 



15 




During the first complete year of MRS services 
(2003), the 10 MRS counties initiated a response 
within a 72-hour time period for an average of 
90.5%*’ of aU cases, as compared with 93.4% in 
these same counties in 2001. These figures are 
not statistically different from each other, 
indicating that the initiation of the MRS system 
did not result in a change in timeliness of initial 
response. 

Nine MRS counties (excluding Mecklenburg) were 
then contrasted with matched control counties. 
For each county, the change between 2001 and 
2003 in the rate of investigations that were 
initiated within 72 hours of report was computed. 
In the MRS counties, the percent of investigations 
initiated within 72 hours of the report decreased 
by an average of 2.9% from 2001 to 2003, 
whereas this percent increased by an average of 
4.5% for control counties. This finding 
approached significance {p = .07). These 
findings indicate that the introduction of MRS 
did not have a statistically significant effect 
on time to initial response; however, the trend 
toward an increased response time is a signal 
to monitor the response times closely in 2004 
to determine whether or not this trend 
continues. 

Case Decision 

A second aspect of timeliness of response is the 
time taken to complete the investigation/ 
assessment and reach a case decision. By policy 
effective through November 2003, case decisions 
were required to be completed within 30 days of 
the receipt of the report (unless a reason for delay 
was documented). In November 2003, this policy 
was changed to require case decisions within 45 
days of a report. Figure 9 depicts the proportions 
of cases for which decisions were reached wihin 
30 days for each year from 1996 through 2003 
for the mean of the nine MRS coundes, the mean 
of the nine control coundes, and Mecklenburg 
County. 



During 2003, the mean propordon of cases that 
resulted in a case decision within 30 days for the 
10 MRS counties was 62.2%^. In 2001, the mean 
was 67.2%, but this difference is not stadsdcally 
signidcant. Next, MRS coundes were contrasted 
with control counties. MRS counties did not 
signidcandy differ from control coundes in the 
change in dmeliness of case decisions between 
2001 and 2003. These findings indicate the 
introduction of MRS did not have a significant 
effecct on time to case decision. 

Figure 10 depicts the propordons of cases for 
which decisions were reached within 45 days for 
each year from 1996 through 2003 for the mean 
of the nine MRS coundes, the mean of the nine 
control coundes, and Mecklenburg County. 

In the 10 MRS coundes (including Mecklenburg), 
the mean propordon of cases that resulted in a 
decision within 45 days was 82.5%, as compared 
with 83.0%* for the same coundes in 2001. This 
difference is not signidcant. AddidonaUy, MRS 
coundes did not signidcandy differ from control 
counties in the change in timeliness of case 
decisions between 2001 and 2003. 

Finally, within 10 MRS counties, family 
assessment and forensic tracks were compared 
for 2003. Figures 11 and 12 show the propordons 
of case decisions reached within 30 days or 45 
days, respecdvely, for both tracks. Cases in these 
two tracks did not differ in the propordons of 
case decisions reached within the 30-day dme 



* These percentages include data from all 10 MRS counties in- 
eluding Mecklenburg, thus they may differ from the percentages 
in Figure 7. 

' These percentages include data from all 10 MRS counties in- 
cluding Mecklenburg, thus they may differ from the percentages 
in Figure 9. 

^ These percentages include data from all 10 MRS counties in- 
cluding Mecklenburg, thus they may differ from the percentages 
in Figure 10. 
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Figure 7: Average Percent of Reports Responded to within 72 Hours 
Across Time, by Type of County 




1996 1997 1998 1999 2000 2001 2002 2003 

Year 

♦ Control ■ Pilot a Mecklenburg 



Numbers based on all reports and may include multiple investigations for the same child. 



Figure 8: Average Percent of Reports Responded to within 72 Hours 
2003 Piiot MRS Counties 




This chart includes all investigations, which may include multiple investigations for a single child. 

Avg % = average percentage of cases investigated within 72 hours in each category. 

Each county’s proportion was calculated and then the average of the proportions was taken. 

Number = total # of investigations in each category (equal to the numbers in Figure 1) 

This category includes both substantiated abuse and substantiated abuse and neglect 

Source: TimeRespFig8_11_12 
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Figure 9: Average Percent of Investigattions with Case Decisions within 30 

Days 

Across Time, by Type of County 




Numbers based on all reports and may include multiple Investigations for the same child. 



Figure 10: Average Percent of Investigations with Case Decisions within 45 

Days 




Control ■ Pilot —a— Mecklenburg 



Numbers based on all reports and may include multiple investigations for the same child. 
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period. There was, however, a significant 
difference in the proportion of cases that reached 
a decision within 45 days; 84.4% of cases in the 
family assessment track were decided within 45 
days, whereas 80.3% of cases in the forensic track 
were decided in this time period. This significant 
(J) < .05) difference indicates that case decisions 
are reached within the 45-day window for a larger 
proportion of families in the family assessment 
track than in the forensic track. 

Timeliness of Service 

Timeliness of service was investigated using the 
earliest recorded service provided subsequent to 
a report of maltreatment. Both Child Protective 
Services reports and Services Information System 
data stored in the Client Services Data Warehouse 
were utilized in these analyses. A review of the 
data revealed striking differences in service 
provision dates for two counties, Bladen and 
Guilford. In both counties, initial services were 
recorded on the date of report for a large majority 
of cases in 2002 and 2003. Based on anecdotal 
reports, these counties have been recording initial 
services on the date that services were first 
requested, rather than on the date that services 
were first provided. Recent service data from 
Bladen and Guilford are therefore not comparable 
to that of other counties and other years. Dates 
of actual service provision are not currently 
available from these counties; thus, they were 
removed from all timeliness of service analyses 
for this evaluation. 



In 2003, eight MRS counties (including 
Mecklenburg) had an average median time of 
27.3 days between the time a report was made 
and the time a service was first recorded. This 
time is not significantly different from the 2001 
timeliness of service in these same counties 
(median = 26.9 days)^. Similarly, pilot counties 
and matched controls show no significant 
differences across years in median time to first 
service. In 2003, the matched control counties 
had an average median time of response of 28.7 
days. 

A second method used to examine timeliness of 
service was to compare the proportion of cases 
that received their first service no more than 30 
days (i.e., the expected time to case decision) 
following their report dates. This number was 
calculated as the following: of aU cases that had 
at least one service recorded, what percentage 
received their first service within 30 days of 
report. The average proportions for seven MRS 
counties, seven control counties, and 
Mecklenburg County are provided in Figure 14. 

For those cases that received services in eight 
MRS counties, 60.9% received their first services 
within 30 days of their report dates in 2001, and 
61.1%'® received their first services within 30 
days of their report dates in 2003. Additionally, 
pilot MRS counties and matched control counties 
showed equivalent proportions across years of 
clients receiving services within 30 days. 



First, the median length of time from the report 

date to the date of first service in each MRS 9 „, . , , , ^ , i- 

X hese percentages include data from eight MKd counties including 
county was examined. The median length of time Mecklenburg, thus they may differ from the percentages in Figure 
represents the time by which half of the cases 13. 
have received their first service. The average 

median response time for seven MRS counties, percentages include data from eight MRS counties includ- 

ing Mecklenburg, thus they mav differ from the percentages in 

seven control counties, and Mecklenburg County p^g^re 14 
are provided in Figure 13. 



Center for Child and Family Policy 



19 




Figure 13: Median Number of Days from Report to First Service 

Across Time, by Type of County* 




Numbers based on all reports and may Include multiple Investigations for the same child. 
*Bladen and Guilford, along with their matched control counties, are excluded from this figure. 



Figure 14: Percent of Reports with First Service within 30 Days 

Across Time, by Type of County* 




Numbers based on all reports and may Include multiple Investigations for the same child. 
"Bladen and Guilford, along with their matched control counties, are excluded from this figure. 
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In 2003, the matched counties provided at least 
one service within 30 days of report to 57.8% of 
cases. Taken together, these findings indicate that 
the initiation of MRS is not associated with a 
change in timeliness of service. 

Within MRS counties in 2003, there were no 
significant differences in timeliness of service 
between family assessment and forensic tracks. 
Cases in the family assessment track had an 
average median time of 26.5 days between report 
and first service, whereas cases in the forensic 
track took a median 27.9 days from report to first 
service. Similarly, 57.0% of cases receiving 
services in the family assessment track had their 
first services within 30 days of report, whereas 
61.8% of cases receiving services in the forensic 
track began services in this time period. 

These findings suggest that the introduction of 
MRS is not associated with a change in timeli- 
ness of service. However, this finding runs 
against the unofficial but consistent focus group 
reports of all nine MRS pilot counties that ser- 
vices are put in place much sooner with MRS. It 
is our speculation that this difference may be at- 
tributed to limitations of the data system in terms 
of how services are coded and entered in the early 
stages of MRS implementation. 

Coordination of Local Human Services 

An important goal of the MRS system is to im- 
prove the coordination of local providers of hu- 
man services on behalf of families in need. It 
was not possible to evaluate change in coordina- 
tion from previous years, nor was it feasible to 
examine coordination in control counties. The 
evaluation of this question was restricted to quali- 
tative analyses within MRS counties, based on 
responses to surveys and focus group meetings. 
The findings should be considered as suggestive 
but not definitive. 



Interagency Collaboration and 
Communication 

All counties made efforts to educate their com- 
munity partners about MRS. Some counties made 
presentations to their community parmers, oth- 
ers distributed fliers, and sdU others had the luxury 
of having county-wide System-of-Care Training, 
which is family-centered practice training for com- 
munity parmers that was sponsored by other ini- 
tiatives. Despite these efforts, most social work- 
ers reported that although communication and 
collaboration across agencies occurred, more of 
it is needed. 

Workers from all agencies spoke of the impor- 
tance of existing interagency relationships with 
their respective community partners and how 
these relationships significantly contributed and 
continue to contribute to their effectiveness in 
meeting the needs of children and families. Child 
and Family Teams were identified as an additional 
forum in which interagency collaboration and 
communication occurs and, in many ways, pro- 
vides a simpler process for this communication 
and collaboration to occur. Community parmers 
further mentioned that CFTs allow them to ob- 
tain a better understanding of the family and its 
functioning and that family members can ask for 
needed resources immediately, when numerous 
agencies are represented. 

The coordination between law enforcement and 
the Forensic Track of MRS varied across coun- 
ties and has not changed with the implementa- 
tion of MRS. Some counties had a Memorandum 
of Understanding signed, whereas other counties 
had more informal relationships with law enforce- 
ment. Some counties mentioned that education 
of law enforcement personnel about MRS had to 
be enhanced. 



Center for Child and Family Policy 



21 




Figure 11: Average Percent of Investigations with Case Decisions within 30 Days 

2003 Pilot MRS Counties 




This chart includes all investigations, which may include multiple investigations for a single child. 

Avg % = average percentage of cases decided within 30 days in each category. 

Each county’s proportion was calculated and then the average of the proportions was taken. 

Number = total # of investigations in each category (equal to the numbers in Figure 1) 

* This category includes both substantiated abuse and substantiated abuse and neglect 

Source: TimeRespFig8_11_12 



Figure 12: Average Percent of Investigations with Case Decisions within 45 Days 

2003 Pilot MRS Counties 




This chart includes all investigations, which may include multiple investigations for a single child. 

Avg % = average percentage of cases decided within 45 days in each category. 

Each county’s proportion was calculated and then the average of the proportions was taken. 

Number = total # of investigations in each category (equal to the numbers in Figure 1 ) 

* This category includes both substantiated abuse and substantiated abuse and neglect 

Source: TimeRespFig8_11_12 
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Collaboration with Work First was inconsistent 
across the counties with some counties having 
systematic ways of including Work First in the 
cases that could use that support and other coun- 
ties not yet including Work First. 

Consumer and Worker Satisfaction 

Because this was a pilot implementation that was 
designed to test the effectiveness of the MRS 
model, the evaluator and the NCDSS decided that 
it would be appropriate to solicit feedback from 
families as recipients of these new services and 
the workers who are those staff members actually 
implementing the new system. As described 
earlier, the evaluation of these questions was 
restricted to either a small sample size or 
qualitative analyses within the MRS counties. 
Therefore these findings should be considered 
suggestive but not definitive. 

Families' Perspective 

170 Family Involvement Surveys were 
administered to families in the family assessment 
track in MRS counties. Responses to these 
surveys were tallied, and the most important 
findings follow. 

- 100% of the families reported that the CPS 

worker treated them in a respectful or very 
respectful way. 

- 99% responded that they were generally 

satisfied or very satisfied with the help 
that they received. 

- 95% of the families agreed or strongly agreed 

with the statement that the CPS Social 
Worker identified strengths in the way that 
they raised their children 

- 98% agreed or strongly agreed that they were 

involved in the decisions that were made 
about their family and children. 

- 42% of families reported participating in 

child and family teams. 



Examples of responses collected from families 
on the surveys include: 

— “If child services had not been involved I don’t know 

what I would have done. Thej bent over backwards 
to help mj family. ” 

— “I just wanted to say that I tremendously appreciate the 

help I received and the respect that I got from my 
social worker. ” 

— “CPS has helped my family to the fullest. They helped 

me build up my confidence in all areas. ” 

— “My social worker not only did her job well, but also 

cared about us and our problems. She did everything 
possible to make sure we could pull through our hard 
time. ” 

— ‘I just wanted to say that they gave me a chance and 

now I made it finally. ” 

— “I think these people are here to help children and to 

help parents realir^e kids come first. ” 

— “My social worker has helped in so many ways I couldn 7 

name them all! She kept me on track and made sure 
everything my family needed was taken care of 
promptly. ” 

— “Our CPS worker was very honest with our needs. 

She seemed very concerned with what we needed! 
wanted. She’s the best!!” 

— “My social worker was my inspiration!” 

— “My social workers were very kind and didn’t assume I 

was guilty of what I was accused of. ” 

In sum, families in the family assessment track 
reported high levels of satisfaction with the 
services that they received. Moreover, families 
acknowledged that social workers were applying 
the family-centered, strengths-based principles of 
the family assessment, with the exception of child 
and family teams. These findings must be 
tempered by the possibility that families that 
participated in these surveys were a biased 
subgroup, selected by caseworkers and self- 
selected to complete a survey. In order to reach 
more definitive conclusions, more thorough and 
objective sampling of families would be required. 
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Figure 13: Median Number of Days from Report to First Service 
Across Time, by Type of County* 




Numbers based on all reports and may include multiple investigations for the same child. 

*Bladen and Guilford, along with their matched control counties, are excluded from this figure. 

Source: TimeServFig13_14.sas 



Figure 14: Percent of Reports with First Service within 30 Days 
Across Time, by Type of County* 




Numbers based on all reports and may include multiple investigations for the same child. 

*Bladen and Guilford, along with their matched control counties, are excluded from this figure. 

Source: TimeServFig13_14.sas 
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County Department of Social 
Service Workers' Perspective 

During focus groups, county DSS workers were 
asked to compare working within MRS to the way 
that they had done business prior to its 
introduction. Both social workers and supervisors 
reported that the family-centered practice of MRS 
allows them “finally” to do social work the way 
that they were taught to do it in school. They felt 
that MRS trains workers to treat families the way 
they themselves would like to be treated. Workers 
identified the following points as positive 
contributions of MRS: 

For the families, 

a) MRS is more respectful to the families than 
the traditionally used approach. 

b) MRS makes the families more open, less 
defensive and less resistant. 

c) With MRS, the social worker takes into 
consideration the whole situation and not 
just the specific incident. 

d) MRS offers families many needed services 
that they would not otherwise receive. 

e) MRS allows social workers and 
supervisors greater flexibility in decision- 
making and service delivery. 

1) Child and Family Teams (CFTs) allow 
families the opportunity to take charge of 
their family and to work with other 
professionals to meet their existing needs. 
In an environment of mutual respect and 
honesty, families play a major role in 
developing a treatment plan for their 
family. Social workers spoke of families 
leaving CFTs with a clear direction and a 
feeling of accomplishment. 

For the social workers, 

a) Social workers consistently reported that 
families are more receptive to them under 
MRS than previously. When the families 
realize that social workers are there to 
help rather than to take their children 



away, they are less adversarial, more 
honest, and more appreciative. As a 
result, social workers reported that on a 
personal note their job has become more 
rewarding. 

b) In counties where Child and Family Teams 
were utilized consistently, either 
facilitated by the case manager or an 
external facilitator, CFTs were described 
as “god-sent,” “the resource that 
safeguards safety,” and time savers. With 
all involved parties sitting around the 
same table, social workers found that 
they did not need to make individual 
contacts with agencies. In other counties 
social workers did not have a clear 
understanding of the benefits of CFTs 
or their roles within them and therefore 
were hesitant to utilize them. 

Workers also identified some challenges with 
MRS. At the same time that social workers 
reported that their jobs were more rewarding, they 
also reported high levels of stress because of: 

a) Mixed Caseloads. 

The 2003 Multiple Response System Policy 
and Practice Manual states: '7/ is highly 
recommended that the social worker that conducts 
the family assessment maintain the case throughout 
the provision of In-Home Services. ” This directive 
meant that social workers were asked to play 
the dual role of assessor and case manager. 
In other words, social workers received new 
assessments and investigations that they had 
to serve within the standard response times 
for the type of alleged maltreatment, while 
at the same time providing the mandated 
number of contacts for their case 
management families. As a result, if they had 
scheduled an appointment for case 
management and a new assessment came in, 
they had to cancel or postpone the 
appointment and tend to the new assessment. 
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This problem was reported consistently 
across the counties that implemented mixed 
caseloads. 

b) Work schedules. 

Following family-centered practice, in most 
counties, workers’ schedules changed to 
accommodate family schedules. This change, 
however, led to workers working longer hours 
and weekends, which adversely affected their 
private family lives. Even in counties that 
had implemented flexible hours, the problem 
was not alleviated substantially because 
taking time off in the morning still kept them 
away from their own families in the evening 
when their own families were at home. There 
was some degree of variability in workers’ 
ability to navigate this difficulty, which could 
be attributed to individual time management 
skills and ability to negotiate mutually agreed- 
upon meeting times with families. 

c) Child and Family Teams. 

Scheduling CFTs can be time consuming. In 
cases when numerous agencies and family 
members need to be invited, finding a time 
that is convenient for everyone may prove 
challenging. The facilitation of CFTs was 
also stressful for some counties because 
outside facilitators were not available. 

d) Feelings of Frustration. 

High mixed caseloads and difficulty 
managing their schedules led workers to feel 
that they were “always treading water” and 
“putting fires out all the time.” Consistendy 
across counties, workers felt they were not 
serving their ‘in-need’ families to their 
satisfaction. 



e) Supervision. 

In some counties, workers stated that 
supervision varied across teams, leading to 
inconsistent criteria in decision-making and 
a feeling of frustration among workers. For 
example, in one team a family might receive 
a case decision of “services recommended,” 
whereas in another team a similar type of 
family might receive a case decision of 
“services needed.” Such inconsistency 
seemed to emanate from two sources: (1) the 
intrinsic ambiguity involved whenever a new 
system is developed and definitions of terms 
are created; and (2) variations in supervisor 
background and training. It is possible that 
these inconsistencies will dissipate over time, 
although better training of supervisors and 
staff workers might prevent these problems 
in new MRS counties and improve 
consistency within and across MRS counties. 

Cost Effectiveness 

Utilization and Re-Allocation of Staff 
Resources 

A formal cost-effectiveness analysis of MRS was 
originally requested by the Legislature but was 
not possible due to the lack of randomly-assigned 
control counties, complexity of fiscal records, the 
brief timeframe stipulated for the completion of 
this report, and the lack of resources. Because 
no additional funds had been allocated for the 
implementation of MRS, the “official” costs of 
implementing MRS were identical to costs in 
control counties and identical to costs in previous 
years in the same counties. Therefore, it was 
decided in consultation with the Division of 
Social Services that a study of the utilization and 
re-allocation of staff resources would serve a 
similar purpose. 
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During the focus groups, social workers and 
supervisors reported that the approaches used in 
MRS implementation were based on worker 
preference, worker style or inclination, or 
geographic assignment. Implementation also 
differed because some counties “jumped in” with 
minimal preparation, whereas other counties 
spent extensive periods planning for the change. 

Despite the different modes and speed of 
implementation, all counties had to re-allocate 
their staff members to fill the needs of MRS. 
Workers were pulled from foster care, 
investigations, case management, or on-call 
duties to staff the new positions. For the most 
part, staff members were re-allocated in one of 
the following two patterns: 

- The role of investigator/ assessor was kept 

separate from the case manager role, or 

- The same person kept the case from 

assessment to service delivery and case 

management. 

A common difficulty in most of the counties that 
used the second approach was that the number 
of cases that the worker carried affected his or 
her ability to serve a family as an assessor and 
case manager at the same time. For example, if 
a social worker managed seven cases for which 
she or he needed to meet weekly, that worker 
had seven appointments each week. If, however, 
a valid report came in that needed to be assessed 
or investigated within 24 hours, one or more of 
these appointments would have be postponed or 
cancelled. It is not uncommon for a social worker 
to have seven case management cases and seven 
assessment cases on a caseload. Workers report 
that having dual responsibility for case 
management and assessment/investigation 
adversely affects the quality of care that they can 
provide to their families. 



At the supervisory level, similar changes were 
made. In some counties, supervisors had to be 
cross-trained to be able to supervise all workers. 
Most supervisors reported spending more time in 
meetings, providing direct supervision or leading 
team meetings and case staffings. One of the 
most time-consuming new roles that supervisors 
had to play was to become facilitators for Child 
and Family Teams. Most counties reported that 
staff members initially did not feel comfortable 
running the Child and Family Teams, although 
some counties (specifically, those with previous 
experience with System of Care or similar 
programs) were comfortable with the staff 
members running the CFTs. In the case when 
the staff member was not comfortable, 
supervisors were trained to run CFTs. Only one 
county reported having an external facilitator for 
CFTs. 

There were no perceived changes in the turnover 
rates due to MRS, which, unfortunately, remain 
high because of the high stress of the job. More 
objective tracking and analysis of staff turnover 
rates in MRS and control counties in 2004 is 
recommended. 
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Conclusions 

Safety 

The introduction of MRS has not altered the level 
of children’s safety in the brief period of its 
implementation, according to official records of 
child maltreatment investigations and 
substantiations. 

Timeliness of Response 

The introduction of MRS has not significantly 
altered the likelihood that families will receive 
an initial response within 72 hours or the 
likelihood that they will receive an official case 
decision within 30 days. There was, however, a 
trend toward slightly increased initial response 
time in MRS counties. Response times in 2004 
should be closely monitored to determine whether 
or not this trend continues. 

Timeliness of Service 

The introduction of MRS has not altered the 
median length of time from report to the initiation 
of services or the proportion of families that 
receive initial services within 30 days of a report. 

Coordination of Local Human Services 

MRS allows for better coordination and 
communication across agencies. Having all 
agencies that provide services to a family around 
the same table with the family, engaged in an 
honest discussion of strengths and needs, allows 
families to take charge and feel heard, respected 
and supported. 

All MRS counties express a need for more 
community partner education. Increased training 
for staff members and supervisors in the 
administration of Child and Family Teams is 
necessary in order to relieve staff stress levels 
and improve quality of service delivery. 



The majority of families in the family assessment 
track of MRS counties are very satisfied with the 
way that their family is treated by the CPS worker 
and with the help they receive. Families state 
that social workers identify strengths in the way 
that they raise their children and involve them in 
decisions about their family. Social workers and 
supervisors unanimously agreed that the Multiple 
Response System is a good way to serve families 
who allegedly maltreat their children. 

Workers appreciated the fact that MRS allows 
them to be more respectful of families and to 
take into consideration the whole picture instead 
of concentrating only on the specific alleged 
incident. They feel that they are able to serve 
the families better and that families could feel 
and appreciate the difference. On the other hand, 
workers report high levels of stress because of 
high mixed caseloads and changes in schedule. 

Utilization and Re-allocation of Staff 
Resources 

In the short run, all MRS counties were able to 
re-allocate staff members and resources to 
accommodate the needs of the Multiple 
Response System without additional funds or a 
change in turnover rates. However, staff 
members experienced increased challenges in 
managing cases while working with new reports, 
leading to increased stress levels. It is possible 
that increased stress will lead to increased staff 
turnover or declining quality of service over time. 
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Recommendations 



Training 



The following are recommendations based upon 
the findings of this reports and feedback received 
through the focus groups in the nine counties 
visited: 

Division of responsibilities 

MRS practice recommends that the same person 
would serve a family from beginning to end 
(assessment and case management to case 
closure). Across all the counties that implemented 
this approach, workers in the family assessor/ case 
manager role had to put off their work with the 
families receiving case management to attend to 
new reports, resulting in stressed workers and 
disgruntled families. 

In conversation with line staff and supervisors, 
the solution offered to this dilemma is to decrease 
caseloads to an average of six to eight families 
per worker and limit case management cases to 
four per worker. 

If the resources do not exist to implement this 
significant reduction in caseload size to allow for 
the “blended caseload”, we then recommend that 
a “team model” be implemented wherever 
possible, especially in the larger CPS departments. 
The formation of teams consisting of 
investigators/ assessors, case managers and foster 
care workers that act as a cohesive unit to serve a 
family is an efficient and effective way to organize. 

Caseloads 

The process and equation that determines 
caseload size needs to account for turnover, leave, 
vacation, etc. Across several counties we heard 
that caseload sizes ranged from 12 to as high as 
25 to accommodate the lack of available staff 
due to the above-mentioned circumstances. 



Training both across and within counties was not 
consistent. Where MRS seemed to be 
implemented effectively, staff was most 
supported and satisfied and supervisors were 
consistent in the application of policies; training 
was a consistent and ongoing process. 

Training should be mandatory for all supervisors 
and social workers, and it should be ongoing and 
available as a component of orientation for all 
new employees. Training should take into account 
that some counties have received numerous other 
programs that may have laid the groundwork for 
MRS, but others have not. 

Training should address the criteria used in 
reaching case decisions within the family 
assessment track to ensure consistency across 
staff and counties. 

Management 

Supervision and management both across and 
within counties was not consistent. In several 
counties, staff reported that a case decision would 
have been made differently by different sets of 
supervisors, and that staff were able to schedule 
their time differently based upon each respective 
supervisor. We recommend that supervisory and 
management training be implemented 
immediately and followed by the ongoing delivery 
of additional training modules to further support 
supervisors and managers. We further 
recommend that as individuals are promoted to 
supervisory roles they complete a series of 
management training sessions. 

In addition, we recommend that each county 
agency review its respective policies and deci- 
sion-making rubrics to ensure that they are con- 
sistent across all teams and managers. Consis- 
tent application of policies and decision-making 
rules makes not only for higher quality services 
for families but also better worker morale. 
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Implementation process 

Again, where MRS was most successfully 
implemented, there was a readiness quotient that 
existed either due to similar initiatives or due to 
long and hard efforts of the management team to 
ensure that there was buy-in, understanding and 
commitment at every level both within the agency 
and with its respective community partners. We 
recommend that a process be developed that 
guides each county through this initial process for 
the purposes of achieving this “readiness 
quotient”. This process should include open two- 
way communication and the involvement of staff 
in all levels of planning and implementation. We 
further recommend that each agency appreciate 
and be prepared for the increased resources 
required in terms of supervision, staff and time 
of managers during the transition, as this is truly 
a systems change and not simply another initiative. 

Community Partners 

All community parmers including school personnel, 
mental health, public health, juvenile court, and law 
enforcement need to be educated and trained as well 
as jointly develop an acknowledgement and 
commitment that these are “all of our kids” and that 
it works best when we work together. This process 
should include all levels of management and line 
staff. We recommend that state dollars be 
appropriated to conduct cross-agency family- 
centered training for all child-serving agencies. 

Policy guidelines 

Greater clarity is needed regarding the use and 
documentation of services provided during the 
30-day case decision process. There seemed to 
be discrepancies among counties and changes to 
policies over time as to what the case decision 
should be for families who receive services during 
the Assessment period and do not need additional 
supervision beyond the 30 days. Some counties 
began using the case decision of ‘services 
recommended’ to document the work and services 



provided during the 30-day period. We, therefore 
recommend the use of a new decision category 
in addition to three options now available, called 
“services received, no further services 
recommended.” 

A system needs to be implemented to consistendy 
communicate across all counties all “questions 
and answers” that arise during the 
implementation so that practice is consistent 
statewide (e.g., FAQ document on the web page). 

Child and Family Teams 

AU line and supervisory staff as well as commu- 
nity partners should receive mandatory training 
on the benefits and value of CFTs. All social 
workers and supervisors should receive training 
in CFT facilitation. External facilitators should 
be available to facilitate certain team meetings 
or provide assistance and coaching. 

Work First 

Few counties have had success in integrating 
Work First and MRS. To accomplish this 
integration, we recommend that Work First be 
involved from the beginning. 

Documentation 

All of the statistical analyses presented in the 
report were based on the official data each county 
enters. It is thus extremely important for those 
data to be accurate. It is our fear that even though 
there is consensus across counties that MRS is a 
positive change and families appreciate it, 
unreliability and inaccuracy in the raw data may 
hinder a valid evaluation. We recommend that the 
Division, in partnership with the counties, 
develop strategies to ensure the accurate 
collection and timely transfer of these data. 
Specifically, we recommend the development of 
an automated child welfare case management 
system. This system would allow workers to enter 
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data in a consistent and reliable way across the 
state. This would also enable counties to 
communicate more effectively. AU staff should 
receive training in the operation of this system. 

Each county should identify one person who will 
be a specialist in the use of the system and will 
be responsible for all data review and for 
questions regarding coding issues. All such 
individuals could be part of a chat room where 
these issues can be openly discussed across 
counties so consistency can be ascertained. They 
should also be responsible for educating the 
remaining staff about coding changes. 

When data are entered in more than one database 
this person should be responsible for 
crosschecking the data and running simple 
statistics to confirm the accuracy of the data. This 
will in turn ensure that when the data are received 
by NCDSS, a basic level of data cleaning has 
already been accomplished. 

Evaluation 

To truly evaluate the impact of MRS, a more 
systematic and detailed evaluation is needed when 
MRS has been in practice for a longer period of 
time. We further recommend that evaluation of 
the next wave of implementation begin 
immediately and be ongoing. This will ensure that 
counties are given timely feedback about their 
practices. It will also provide a timeframe longer 
than six months to examine the rates of 
reinvestigation. Such an evaluation should be 
two-fold: the 10 pilot MRS counties should be 
evaluated by themselves at the end of 2004 and 
compared across time. Unfortunately, 
comparisons with control counties will not be 
possible for the 2004 year because some of the 
control counties received MRS in the new wave 
starting in October 2003. The new 42 counties 
should be evaluated across time and if possible 
compared to appropriate controls. 



A more in-depth evaluation may include: 

a) Focus groups with families in both the 
Family Assessment and the Forensic tracks. 

b) Focus groups with families who received 
services before and after MRS was 
implemented. 

c) Focus groups with front-line workers. 

d) Focus groups with supervisors. 

e) Focus groups with community partners. 

f) Family Involvement Surveys with 
randomly chosen families from both tracks 
and in both the assessment and case 
management phases. 
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Data Sources and Data Processing 
Child Protective Services (CPS) Reports 
Source 

Data provided in the Central Registry records of the Client Services Data Warehouse are 
from the DSS-5104 form. For this evaluation, data were extracted with the following parameters: 

Source Data Month — December, 2003 (“200312”). 

Time Period — 1996-2003. Records from 1/1/1996 through 12/31/2003 (inclusive) were 
selected based on the Report Date. 

County — County Name was used to select data only for the 10 pilot and 9 comparison counties. 
View — All fields are selected from the Central Registry Victim View. 

Fields — The following fields were included: 



Initial Report Date 


Investigation Initiated Date 


Investigation Completed Date 


County Case Number 


Form Number 


County Name 


First Name 


Middle Initial 


Last Name 


Birth Date 


Race 


Sex 


SISCHentID 


Social Security Number 


Source Data Month 


Type Reported 


Type Reported Code 




Type Found 


Type Found Code 




Maltreatment 


Maltreatment Code 




Risk Assessment Rating 


Risk Assessment Rating Code 





Processing 

Initial Processing 

The 19 data files were downloaded from the Data Warehouse, and converted into a SAS 
dataset via SAS programs. This process included re-naming variables, converting dates to SAS dates, 
converting “#EMPTY” values to blanks, and other non-substantive changes. In addition, a unique 
ID was assigned to all records for each child according to the following rules: 

1 . Records in the same County with the same SIS # are assigned the same ID, AND 

2. Records in the same County with the same Last Name, First Name, Birth Date, and Sex 
(where all values for these fields are non-missing) are assigned the same ID. 

There were a total of 325,664 records (all 19 counties, all years 1996-2003). 
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"Fuzzy" Matching 

The data were further processed to assign the same unique ID to records with slight variations in the 
Last Name, First Name, Birth Date, or Sex fields. In all cases, the records were required to be within 
the same county, and the identifying fields were required to be non-missing. In some cases, SSN or 
the Form Number were used to verify whether variations in the identifying variables indicated the 
records were for different children. 

Duplicate or Multiple Records 

Records showing duplicate or multiple reports within 30 days exist in the CPS data. State policy 
dictates that only one record should be submitted for each report. Furthermore, if multiple reports 
for the same child occur within 30-days of each other, the reports should be combined into one 
report. The county should report only one record to the state, and they are supposed to report the 
most severe case. The severity of the case is based on whether the report was substantiated or not, 
and the maltreatment found for substantiations. Duplicates and multiple records for the same children 
were processed according to state policy as follows: 

/. Complete Duplicates — There are 134 records that are exact duplicates (67 records with an exact 
duplicate second record). While there may be some distinguishing information in the fields 
that are not downloaded from the Data Warehouse, all fields pertinent to MRS are exact 
duplicates. 

Only one of these records is kept. 

2. Children with Multiple Records from the same Vorm — Most of these have the same Case #, Report 

Date, and report information, but different SIS numbers. They have the same ID because 
their Last/First Name, Birth Date, and Sex fields are the same. In some cases, the differ- 
ence is in the report information (type of report, abuse/neglect found, and type of mal- 
treatment). 

Onlj one of these records is kept. Where there are differences in the report information, the record for the 
most severe case is kept. 

3. Records that are Duplicates EXCEPT for Form # - These records are duplicates, and there are 

additional records that are duplicates except for the Form #, SSN, Race, and Middle Initial. 
These records have the same county, ID, case #, report and investigation dates, risk rating, 
and report information. 

Only one of these records is kept. 

4. Children with Multiple Rxcords bj County, ID, and Keport Date — This list does not include any 

records from (2) and (3) above. In aU cases, the Form # is different. For 65% of the cases, 
the Investigation Start/End dates are the same. 78% of the time, the Case # is the same. 
48% of the time both the Investigation Start/End dates AND the Case # are the same. 
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5. Children with more than one report within 30 days — These are children with multiple incidences 
within 30 days. 

All reports within 30 days are combined into one record. Each field is looked at separately, and the 
worst-case for the field is kept in the combined record. 

Final Data File 

The final data file contains 306,124 records. The final SAS programs to process these data 
are as follows: 

ReadPilot 01/08/2004 12:04:27 PM 

ReadControl 01/14/2004 01:00:06 PM 

lDl_lnit 01/14/2004 02:04:59 PM 

lD2_Fname 01/19/2004 08:02:38 AM 

lD3_LName 01/19/2004 08:20:57 AM 

lD4_BDate 01/19/2004 08:34:58 AM 

lD5_Sex 01/19/2004 08:42:37 AM 

CrMastCPS_9603 01/19/2004 08:57:03 AM 

CleanCPS 01/30/2004 11:14:51 AM 

Services Information System (SIS) Data 

Source 

Data provided in the SIS records of the Client Services Data Warehouse are from the DSS- 
5027 form. For this evaluation, data were extracted with the following parameters: 

Eeport Month — December, 2003 (“200312”). 

Time Period— 1996-2003. Records from 1/1/1996 through 12/31/2003 (inclusive) were 
selected based on the Service Begin Date. 

County — County Name was used to select data only for the 10 pilot and 9 comparison 
counties. 

t/iew — AU fields are selected from the SIS View. 

Fields — The following fields were included: 

Service Begin Date Service Terminated Date Application Date 

Termination Date County Name County Case Number 

Client First Name Client Middle Initial Client Last Name 

Client Birth Date Race Sex 

SIS Client ID Client SSN Report Month 

S ervice Code Service N ame 

Service Reason Code Service Reason Description 
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Processing 

Initial Processing 

The 19 data files were downloaded from the Data Warehouse, and converted into a SAS 
dataset via SAS programs. This process included re-naming variables, converting dates to SAS dates, 
converting “#EMPTY” values to blanks, and other non-substantive changes. There were a total of 
562,189 records (all 19 counties, all years 1996-2003). 

Assignment of ID 

The data were further processed to assign the same unique ID to all records for the same 
child. Within each county, SIS records were matched to CPS records by SIS number, and again by 
Last Name, First Name, Birth Date and Sex. Since the SIS data include information for adults and 
children who receive services unrelated to CPS reports, only SIS records for children with a CPS 
report were kept for further processing and analysis. 

Final Data File 

The final SIS data file includes information from both the SIS and CPS data systems. One of the 
services included in the SIS data is a “CPS - Investigative Assessment” (service code “210”). Where the 
SIS data contained multiple “210” records for the same child, only one record was kept. Then, the SIS 
“210” records were combined with CPS report records by child and date within county. AH CPS records 
with no corresponding SIS code “210” record were also kept, along with all non-“210” SIS service records. 

The final data file contains 492,803 records, and are broken down as follows: 



248,857 Combined SIS “210” and CPS report records 

32,478 SIS “210” records with no corresponding CPS record 

57,267 CPS records with no corresponding SIS “210” record 

154,201 SIS service records (non-“210” records) 

The final SAS programs to process these data are as follows: 



ReadPilotSlS 

ReadControlSlS 

CrMastSlS_9603 

CleanSlS 

Focus Groups 



01/22/2004 

01/22/2004 

01/22/2004 

01/30/2004 



09:37:19 AM 
09:40:58 AM 
09:51:40 AM 
11:27:19 AM 



Sample questions for front-line workers and community partners: 



a) How has MRS been able to serve families differently from before? 

b) How has the introduction of MRS affected your job? Time-wise, satisfaction-wise. How 
have your duties changed? 

c) What aspects of MRS work the best in your opinion and why? What aspects do not work 
very well and why? 

d) How has MRS affected coordination among agencies? Give positives and negatives. 
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e) How do Child and Family Teams work and how have they affected coordination among 
agencies? Give positives and negatives. 

Sample questions for supervisors: 

a) How did you implement MRS? (shifted focus and orientation, shifted staff and supervisors) 

b) What happened to social workers who were investigators? Were they retrained? How 
long did it take? What was the cost? How was the turnover? 

c) How much time have SW spent in court before and after MRS was introduced? 

d) How has your personnel turnover been for those working within the MRS dual track 
approach vs. the Investigative one? 

e) What difference do you see in the way families are served under the MRS dual track 
approach as compared to the way things were done before? 

1) How are caseloads figured out within the MRS dual track? 

g) How do Child and Family Teams work? 
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Data Analyses and Statistical Findings 

To investigate child safety, timeliness of response, and timeliness of service in counties 
piloting MRS, Child Protective Services reports and Services Information System data were used. 
For each analysis, data trends over the past five years were graphed and examined to ensure that 
control and pilot counties were following similar trajectories unfil 2002, when MRS was preliminarily 
initiated. In all cases, trajectories were equivalent, thus a single comparison year was used. As the 
most recent year with no MRS usage, 2001 was selected as the non-MRS comparison year. Data from 
2002 were not utilized, as MRS was not yet in fuU usage across pilot counties. 

Data Preparation 

Safety 

To examine child safety, the proportions of all investigations (both forensic and family 
assessment combined) that were substantiated as abuse were examined. The average proportion 
across pilot counties was computed using the mean of the county proportions; proportions were 
weighted by sample size, thus each county contributed equally to the overall mean. Abuse was 
chosen as a comparison, because abuse cases should remain in the traditional forensic investigation 
track even in MRS counties, and one would expect that in the first year of MRS, rates of abuse 
should not have changed. Traditional “substantiations” do not occur in the family assessment track, 
and treatment need decisions are based on somewhat different criteria in the two tracks, thus neglect 
rates are less comparable across years. 

Second, safety was examined as a function of the rates of re-investigation (“investigation” in 
these analyses refers to both forensic track investigations and family assessments). The number of 
children with investigations in the first half of each calendar year (i.e., January 1-June 30) was calculated 
for each county. Then, the percentage of these children who had a second investigation within 6 
months of the first was computed. Only children with initial investigations in the first half of the 
year were used, because 6-month follow-up data were not available for children investigated in the 
second half of 2003. 

Third, the number of children with substantiated abuse cases in the first half of each calendar 
year was calculated for each county. Substantiated abuse was chosen because it was handled 
consistently across years and across counties; even with MRS, abuse cases are investigated and 
substantiated following the traditional child protective approach. Therefore, numbers and types of 
cases substantiated as abuse should be comparable. The percentage of children with a substantiated 
abuse case who had another investigation within 6 months of their substantiation was computed. 

Timeliness of Response 

For timeliness of response, the most appropriate variable to examine was the percentage of 
cases that met the mandate for length of time between report and initial contact. Mandates vary by 
type of report, however, with some cases being flagged as requiring immediate response, some 
requiring response within 24 hours, and some within 72 hours. Using the data available, however, 
there was no way to reliably determine the mandated response period for each case. Because aU 
cases should be responded to within 72 hours, this was the unit of time investigated in these analyses. 

To prepare the data for analysis, the length of time between report and initial contact was 
first calculated. The proportion of cases responded to within 72 hours was then calculated for each 
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county. Additionally, the proportions of cases that reached a case decision within 30 days and within 45 
days were calculated. Again, these were calculated within each county. The average proportion afross pilot 
counties for each of these analyses was computed using the mean of the county proportions. 

Timeliness of Service 

The time from the report date to the first service provision was calculated, and the distribution of 
this data within each county was examined across years. Extreme differences in the method of recording 
first services were identified for Bladen and Guilford counties. Specifically, with the initiation of MRS, 
these counties began coding services (“at risk case management services” or “protective services for 
children — team setting”) on the date of the report for the majority of cases. Because these counties coded 
service types and dates differently following the initiation of MRS, their services data is not comparable to 
that of other counties or across years. For this reason, Bladen and Guilford counties, along with their 
matched control counties, were excluded from all analyses related to timeliness of service. 

For the remaining counties, two timeliness of service variables were calculated to be used in 
analyses: (1) median number of days between report and first recorded service, and (2) percent of cases 
that received at least one service within 30 days (expected time of case decision) of the report date. 
Medians were used instead of means due to the presence of extreme outliers in the data. These outliers 
caused means to be shifted upwards, thus the median provided a better description of central tendency. 
The average median/proportion across pilot counties for each of these analyses was computed using the 
mean of the county medians/proportions. 

Statistical Methods 

For the safety construct, data were analyzed in two ways. First, data from the 10 pilot MRS 
counties were compared for the years 2001 and 2003 to look for changes over time within these counties. 
Paired t tests were used (matching by county) to determine whether changes were significant. This statistical 
test was chosen for simplicity and to increase statistical power. Nonsignificant findings indicate no change 
over time. 

Second, changes from 2001 to 2003 were compared between the nine MRS pilot counties (without 
Mecklenburg and nine matched control counties to see if pilot counties changed differently (more or less 
than) than control counties over time. For each county, the changes between 2001 and 2003 were computed, 
and the average change for MRS counties was contrasted with the average change for control counties 
using a paired t test. Nonsignificant findings indicate that any changes over time in MRS counties were 
equivalent to changes over time in control counties, thus MRS did not affect the given construct. 

Descriptive data were provided on the differences between the family assessment and forensic 
tracks, but the tracks were not compared statistically on the safety construct due to the large difference in 
cases assigned to the two tracks. Cases in the forensic track include those with more serious risk of harm, 
thus safety would not be expected to be comparable across tracks. 

For each timeliness construct, data were analyzed using the first two methods described above. 
Additionally, family assessment and forensic track cases were compared within pilot counties in 2003 to 
see if there were differences across tracks. Proportions were calculated separately for cases in each track 
within each MRS county. Again, paired t tests (matching by county) were used to assess significance. 
Nonsignificant findings indicate that no differences between the forensic and family assessment tracks. 
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Statistical Findings 

Findings from the paired t tests are presented in the tables A1 through A3 below. 



Table Al. T Tests for Child Safety 



Variable 


Comparison 


Statistical Test 


Statistic 


% of investigations 
substantiated as abuse 


Pilot counties 2001 to pilot counties 
2003 


Paired t test 


/(9) = 1.79, ns 


% of investigations 
substantiated as abuse 


Difference btwn 9 pilots and 9 
controls in 2001 to difference btwn 
9 pilots and 9 controls in 2003 


Paired t test 


/(8) = -0.05, ns 


% of investigations 
re -investigated within 
6 months 


Pilot counties 2001 to pilot counties 
2003 


Paired t test 


/(9) = 0.72, ns 


% of investigations 
re -investigated within 
6 months 


Difference btwn 9 pilots and 9 
controls in 2001 to difference btwn 
9 pilots and 9 controls in 2003 


Paired t test 


/(8) = 0.26, ns 


% of abuse 
substantiations 
re-investigated within 
6 months 


Pilot counties 2001 to pilot counties 
2003 


Paired t test 


<9) = 1.32, ns 


% of abuse 
substantiations 
re-investigated within 
6 months 


Difference btwn 9 pilots and 9 
controls in 2001 to difference btwn 
9pilots and 9 controls in 2003 


Paired t test 


/(8) = -0.54, ns 
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Table A2. T Tests for Timeliness of Response 



Variable 


Comparison 


Statistical Test 


Statistic 


% w/ initial contact 
within 72 hrs 


Pilot counties 2001 to pilot counties 


Paired t test 


/(9) = 1.90, ns 


% w/ initial contact 
within 72 hrs 


Difference btwn 9 pilots and 9 
controls in 2001 to difference 
btwn 9 pilots and 9 controls in 2003 


Paired t test 


7(8) = -2.06, p=.07 


% w/ initial contact 
within 72 hrs 


Family assessment track in pilot 
counties 2003 to forensic track in 
pilot counties 2003 


Paired t test 


f(9) = -4.06,p<.01 


% w/ initial contact 
within 72 hrs 


Abuse cases in pilot counties 2001 
to neglect and dependency cases in 
pilot counties 2001 


Paired t test 


t(9) = -2.77,p<.05 


% w/ case decision 
within 30 days 


Pilot counties 2001 to pilot counties 
2003 


Paired t test 


7(9) = 1.05, ns 


% w/ case decision 
within 30 days 


Difference btwn 9 pilots and 9 
controls in 2001 to difference btwn 
9 pilots and 9 controls in 2003 


Paired t test 


7(8) = -1.26, ns 


% w/ case decision 
within 30 days 


Family assessment track in pilot 
counties 2003 to forensic track 
in pilot counties 2003 


Paired t test 


7(9) = 1.71, ns 


% w/ case decision 
within 45 days 


Pilot counties 2001 to pilot counties 
2003 


Paired t test 


7(9) = 0.15, ns 


% w/ case decision 
within 45 days 


Difference btwn 9 pilots and 9 
controls in 2001 to difference btwn 
9 pilots and 9 controls in 2003 


Paired t test 


7(8) = -1.09, ns 


% w/ case decision 
within 45 days 


Family assessment track in pilot 
counties 2003 to forensic track in pilot 
counties 2003 


Paired t test 


t(9) = 2.29,p<.05 
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Table A3. T Tests for Timeliness of Service 



Variable 


Comparison 


Statistical Test 


Statistic 


Median number of days 
from report to first service 


Pilot counties 2001 to pilot counties 
2003 


Paired t test 


<7) = -0.19, ns 


Median number of days 
from report to first service 


Difference btwn 9 pilots and 9 
controls In 2001 to difference 
btwn 9 pilots and 9 controls In 2003 


Paired t test 


liG) — 1.62, ns 


Median number of days 
from report to first service 


Family assessment track In pilot 
counties 2003 to forensic 
track in pilot counties 2003 


Paired t test 


<7) = -0.35, ns 


% w/ first service 
within 30 days of report 


Pilot counties 2001 to pilot counties 
2003 


Paired t test 


/(7) = -0.05, ns 


% w/ first service 
within 30 days of report 


Difference btwn 9 pilots and 9 controls 
In 2001 to difference btwn 9 pilots 
and 9 controls In 2003 


Paired t test 


<6) = -1.24, ns 


% w/ first service within 
30 days of report 


Family assessment track In pilot 
counties 2003 to forensic track in 
pilot counties 2003 


Paired t test 


/(7) = -0.78, ns 
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